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(54) Title: 



MULTI-VIEW IMAOE REGISTRATION WITH APPUCATION TO MOSAICING AND LENS DISTORTION CORREC- 
TION 



(57) Abstract 

A system and process for true multi-image 
alignment that does not rely on the measurements 
of a reference image being distortion free. For 
instance, lens distortion is a common imaging 
phenomenon. When lens distortion is present, 
none of the images can be assumed to be ideal. In 
an embodiment of the invention, all the images are 
modeled as intensity measurements represented in 
their respective coordinate system, each of which 
is related to a reference coordinate system through 
an interior camera transfonnation and and an 
exterior view transformation. Motion patameters 
(16, 17, 18) determined in accordance with an 
embodiment of the invention dictate the position of 
the input frmtes (1 1. 12. 13) within the reference 
fmnc (15). A reference coordinate system is used, 
but not a reference image. Motion paiameters (16, 
17, 18) are computed to waip all input images to 
a virtual image mosaic in the reference coordmate 
system of die reference frame (15). Each pixel 
in die virtual image mosaic may be predicted 
by intensities at corresponding pixel positions 
from more than one image. The error measure, 
which is the sum of die variances of predicted 
pixel intensities at each pixel location summed 
over the virtual image mosaic, is minimized. 
The embodiment of the invention advantageously 
maximally uses information present in all images. 



FRAiyiEl 



T 
11 



FRAME2 



FRAMES 




7 

15 



I 

I 

VIHTUAL 
MOSAtC 
REFERENCE 
FRAME 



♦(Refiarod to in PCT Gazene No. 32/1998, Section 0) 



BEST AVA/UBLE COPY 



{ms^mfff (jp) (12) ^ ^ 1^ ^ ^ (A) (ii)»iipaH4^»# 

2^^2002-515150 
(P2002 - 515150A) 
(43)^ B ¥mA¥ 5 ^21 B (2002. 5. 21) 



(51) Intel." 




F I 


f-73-h-(^) 


G0 6T 3/00 


300 


G06T 


3/00 3 0 0 


1/00 


2 80 




1/00 2 8 0 


H 0 4 N 1/387 




H04N 


1/387 


5/262 






5/262 








mn* "jHsmm* m (^ 41 h) 


(21)ii}B»# 


4$BPP10-522833 


mmmA 




(86) (22)ttiBIB 


^JEK 9 ^11^ 13 B (1997. 11. 13) 






(85)ni;:jtflttiB 


^fSil 1^ 5 ^ 13 B (1999. 5. 13) 




'^^)>7sV> '7$'>h> D—H 201 


(86)BK(U0## 


PCT/US 9 7/2 0 7 6 2 




>^X5< 5300 


/Q7\ fiTIMtrflN Miff B 




(72)RWw 




(87)BR(&nB 


^J«10^ 5 ^22B (1998. 5. 22) 






(31)ffjfe4g^S$% 


6 0/0 3 0, 8 9 2 






(32)ajgB 


TJ* 8 ^1 1 ^ 13 0 (1996. 11.13) 




— H 966 


(33)«|jfeti£SS 


*H (US) 


(72)9SI»« 




(31)ffjtsli£S«# 


0 8/9 6 6, 7 7 6 








¥l£ 9 ^1 1^ 10 B (1997. 11. 10) 




i'^^'^U— ^JV'^'OV D— K 17 


(33)Sjfe«i£SB 


*S (US) 


(74)f<SllA 


#ai± [DEB ff- (^1.1«) 











(54) Biiiao«»] ^if-({;^i:x^\^>x^3mjE^<omm\zr3\^x<o^wsLismmm 



(57) Bli|l«Jl 

« <& CD ^ UTSipntto-t&icM e,fs.^^M<o 

^.J*-^' (1 6> 1 7. 1 8) »v Sns^P-A (1 

5) '*'©X*:7k~A (11. 1 2> 13) OttBSit 

n^aVi. Saft/^9;>1-^(1 6,17.1 8)tt. Sip7 
P-A(l 5)«!)»?lJ!^»lc:*V>T, ■T'^TOAABfgl 

ts«^if -f ^ -a-s <fc 3 tf-® $ ns. 

IE«0®{ft^1f'f te:6ft*#-«r©t!^-fe;Kpiiel) 




( 2 ) 0 0 2 -5 1 5 1 

(b) mmmmJ^^m^ ^(esiab\ish)X'r y 

(c) &mmmtLx«fiMim»i<Dm»^m^^i>^tu<, itrE§ttBiofcis«i^ 

(d) mmmm^ nrzmm^wi ^mm mm m\zmf^L , ^^if^^^m^^Lx^T. 

7.7- y y h 

:^ - 9 ^m^'^^^ -XT- y y b. 



( 3 ) 002-5151 50 

(a) if a - )\^ 7 ^ - if ^ m t m r 7. y . 

(b) tfi fR if a - n )i m.m 7 ^ - =t m X - & (o 7 y ^ >/t7/'-^^m^ 

(c) ME — <a©T7-f >/l7Pl-^S:ffl<'iTiti^A7Pt-:5'4o<ttXU>Xli*. 

*«(IA«>i9*«l lCfH«©:^ffi. 

5. ^ iS: © ^ <^ (ins t ruc t i on) & tS -r -5 , a > tf j. — ^' jjt g£ tr C i: 7i« T ^ -5 
«l#T?*oT, ME««!cO#'^rtt, r D -tr >y -y- IC J; Tl^ fr $ n -5 t # . 

(b) a«ffiSJS*M:ftSX5^-y:rt, 

(c) &mmm ii I'.T mmm^omm^ m ^ ^ trj. < . mw.^ifm^rzmm^ 

(d) =E^f-cif^mmr^t:itf>\t. mmmm^ nr^imm^fim-r ^ 7.7- y y t 

6. ^ m m }^ m m <o fc lb <D m m & X . 

(b) a«ffi«S3i5S:»tt-&Ac*e9^et. 

(c) mmmm t vx m m-m. ik<Dmm=s:m^^^z.iit£<. t$im^ifm-Dtimm=s: 



• ( 4 ) !|$^2 002-515150 

8. 2 ^TcUa * ^ 1f -f ^fiRr -5 fc«> ©.:^ft T * o T , 

(c) m mmm 7 - ^ \z ft> ^ X , mmti»<Dmm^mti=e-*f^ if mm^m 

9 . 7.^ y y ( b ) tt . 
(bl)7^ — — A'vrofiyiJt, 

(b2) 7l'-A*^^^r-tf'f;5''^(Dli^Jt 

1 0. WiE2:^7cfi^®MStt, 
» ^ ti * «(f B « » CD B ^ © — o * S -r X 5^ y y t , 

(b) It » o -7 - :/ s n fc® tffc « s*?^ n s a*! tc o T , mmti ^ ti tc 'y 
-y^ntzmm^^^^t, t.^ -j y (a) jc^ji.iXj^jg^nfc'bcD.hUTfifeo 

y M "7 ^ V Y ^f/^^t ^-PsT- y y . 

(d) 7.7- y y (a) IC*5l»Ti*3e$n>S:'bOi:LT-€-n-l?n©|trfE'7-+>i' 
^■If-I'i' y H*^Sttl:^^:-tf'f^ fcf^S V \^ \i i7 )V ^ ^ 7. 7- y Zf 
t , 

(e) «tf|SHi :'3 ^-tf -f e V 5 K «• tJ(col lapse) d t tc J; ^ T , M IE W :/j 

11. 2 jjtTcm*^ if 'f ^a«*^B!c-r-sfc«>©^aT&o T, 



( 5 ) 002-5151 50 

12. WIS 2 ;*C7cJi^«:llff TSfciie>W^&«. 

mmmmou -y^ nttmrno^MiR^ nrz'&mm\z-::>\,^T , tummtR-s ntz'7 
tiim'7-^>{f'i:^-(i7<o^ti'eti\z-D\>^xt}'y'>7>i=^zfyy^z^7>if: 

«tffata:^^:-tf'fi7tf^5iy HSrfcfctrc:<!:{CJ;-pT. tfrEa:^^^:-!!-^^^^:^^ 

1 3. i^>xm^^wmv. wm-snrzu>xm^^mmr^tztf> izm » ■ 

«fe © n n ffl IE T -5 «e) O ft T 3& T > 
(b) U>X^*©;a:tr>^ — OjiS;K^^KrE*|i»©iii«iS:S?iJ-r-5;'S:«>iC/t7pt 

(d) m m ^ tx rz u > X m ^ \z r:> u X m lET ^ fz tb \z tti mm ^ <D ^ ti ^ n r; — 
* # A s ft . 



( 6 ) 002-5151 50 

[ % ^ <0 # SO ift n ] 

^mmit. 1 9 9 6^1 1^ 1 3B\zmm^nft{S.tiim (Provisional Applic 
alion) ^6 0/0 3 0. 8 9 2 ^ \Z ^ -:} < f£\f^ Hi M (non-provisional app 
1 i c a t ton) T % . 

1 . % ^ g> fe^ ^ g 

-S * ©S *P (reference) CD ^ffl JCS at* -5 . 

*i:T-5fej&Jr^^D$n*(v--:^$n-5)c:t*«*«. m*"*J;o:f^ 

^#JS«t'©daST*S1994^1 1^140 tcWS^?nfc ^08/339. 4 
9 1 #(=& fi^ f± SYSTEM FOR AUTOMATICALLY ALIGNING MOSAICS TO FORM A MOSAIC 
IMAGE). 1 996^7^1 7B JCtbSSn it ^60/02 1. 295 #(*«vtiM 
ETHOD AND APPARATUS FOR MOSAIC IMAGE CONSTRUCTION), 1995^6^22B 
JCtbS^tlfc^0 8/ 4 9 3. 6 3 2#(i&|Sx « METHOD AND SYSTEM FOR IMAGE C 
OMPENSATION USING A. PARALLAX BASED TECHNIQUE). *5iUCi996^4^18 
B W S ^ tltl ^60/015. 5 7 7 tt COMPUTATIONALLY EFFICIENT D 

IGITAL IMAGE WARP I NG) »C M * * T * 0 , d tl ^ <D iSC fi* f* # fiS t" -5 ^1 i: J: o 

(3D parallax) :i6 J: m \Z W) < W<D MW} (D ^ v fJ. m M m m ^ M U ^ ^ ^ 

ni^mM^r^tztbizmm^nx^rz. va^Lfjii)^ 

56 M o M 5^ 



(7) «f« 2002-515150 

i7 ^ is. ^ tLT ^ It >!f> \z m ^ ti ^ , 
m 1 \t. *5£wosii6©j^siic«!5, is.m<ommy I'- i^izis» ^^^^ i7 m 

H O 0 3iR T * -5 . 

0 2 «. if 7 5. y H »cS-:^< (py ram id-based) »<a let S (coarse- to- fine) a 
16/^ 5 yf- — <D & ^WlW M (mini avm variance. estinialion)©0:^7?*-5. 

0 4 « , ±^ti. (severe) U>X^*-?&*-r-5*;>< 7iCj;:3TSe)n}tfcf7^:ti^ 

e 



( 8 ) 002-5151 50 

mi 3tt, m 1 2 (o y a - ^ r - h <D 7.7" y y <D J: t) t^muwum^^ ft y a 

O iHI IBilt 8E (bu I t i-image al ignmen t)iC ■:>lr>T©->XT-A*JJ:Of:/D-fe:^-e 

m I m ^ n ^ J: ^ iz , ~ m <o X ti y I' — A I I . 12. i3(^$nsj:'5 
ui,^. ^§tm<ommo)mm\z^-o TViM^ nftmw)/^ ^ — ^ I 6 , 17. is 

«, a2fS7W-Al 5CDrtiB!UC*3lt^A;'37l'-A©te:«*m3£-r^. 

@2$#Mr<&^. 2|s:tf-S[(^, fi^^S^IUKS^Cdirect mu 1 t i -reso 1 u t i on o 
ethod)^^ UTtf^$n;&S^ni»C!>S/h:i)-«l?fli (mini nun 

variance es t i na t e) S ^ < , S iffi ?S fi ffl 6> n -5 *^ , S jfj « ^ 6 

/j:*'*. mw} /-^ y - ^ it , mmy \y - A (ommmn^ izis\,^x . -r ^x <d :K:f]m 

:t3 ^ ^ >tr <D if -ilJV (p i xe \ ) \t , 1 0<fc 0#Viiij^A^e>O^St::J;oT^jB!lT 

#■6. 'tn'en<D\fi7±i}i'<D&.mizi5\,^x'j^m^n^, uw^omm^eif ^ i7 izt> 



( 9 ) 2002-5151 50 

<Dmt&\t. ^«cOg/J>ffiJc5lJS-r«it«i)»C2-5©«B&*ffll»-5. 03 tC.^ ^ tlX 
-5 J; o . ^ — (iPg ® M ^ ^ tt(hierarchica 1 c omp 1 ex i t y)T ^ : — 3 « ^ H 

ic *3 T * a tc ^ . iS IS <d «t» fc is t , H * « n S t* ;^ # < . -en 

f0A. *i«?«coio©siMoj^,®ic*itiT«, -en^'n«iii«!coi^fi!ii/3ws 

© U & n (0 3 O 7. 7^ y r 3 0 1 ) . , D - A* ;i/M ji(global 

translat ion)/'< 7 / - (^laife tC O # 2 -r^)**® # a $ n -5 . 

<D7 y ^ >(affine)/t7;* — (^iSfifelCO^ 6 0)*^Ji)^fc«e){C<seffl$n-5( 
:;^7=-yr3 0 2). -ffl©T7w>/l7;<-4'©«?tt, l*i^>''^7>^-^'(#iaili»C 

5^yr3"0 3). J:©g;ii«;:?.5=-<y7'Ttt, iB^^^d^fflVi^.n-S. 

2 C ^ S n T ^ fiSffi (coarse-to-fine) « T? liS: S tl S . # jS » »C O T . 5 
7 -> T > (laplacian) * tt -> 7 > (gaussian) bf 7 5 !y K ^ ^ $ n -5 , tf 

TS. d © gjift « L M ^ (Levenberg Harquardt t ec hn i que) $ ffl T S n « 

e.n (projected), ± © P -fe X fl «l 0 ig $ tl -5 . Cl©^ft 
^©>^D-fex«^!>;i^<<»:fc2o©fc;ffl^Wr-5. ^ — ©jJiffllciJiiXli, c:© 



( 10 ) #£2 002-5151 50 

©«r3£OiiJISlcft#-r-5ie5.S*<;^c<. (.i \) MM m ft \t - •tr > 7.) iiU ^ <D 
&Vi(0 y U — AtS«l07 \^ — A t (Dm ^zm^3. 0 <D fi l,-^ IKA^ ftlexlended)m 
T&-5<i:#T$>l. iIf|feS:IE®Liffili^tft*tf»-r-5Clt*<T#, (iii)W — © 

jft$S*#T-g)#«. Iti^^tft**t)t3<i:'fc— feWiiJt^xJPTa&O. ^miilft 
(/t > (pan, ± T © # / A ^ 5 O *5 0 ( t i 1 ffi ^ ) ) *5 J; X — A © 

«8ie»c¥ffi8tj^^tft<tl feL<«2^»^>7rii*l5^i:7i>6.;a:-5. ^-nsiDA 

Sll (x' .y' )JcBliS^lt ^ n * . m 1 X 7^ ^{C*5ViT, (x.y)«> ¥ 
©*.JcJ;oXTfB©iCOia(i)tcj;r)SllSi^ffiS(x,',y,')'v^^sn-5. 



( II ) 



!Rra2 002-5151 50 



' <'3)**«32>'*«33 

i^S (x, ' .y, ' ««) * nfc f 5=^ja5g(x' .y')*#S fc* IC. ^ © itft ( 2 ) 

y' =.v', +7|0'/ ->'c>''^ f2) 

:i d TPc ' = (xc ' . Vc ' ) « i # S CO 7 - A tc MT -5 Mi^ 4" 'D-T i& 0 . r' =(x, ' -Xc ' 
)' + (y, ' -Vc ' )' tt tt^-D*^ e» « (X, ' . y, ' )©fgii|0 3p:^T * S . «iflJffiS(x. y) t 

T € * . " 

p' =/>^ +r(p^:y,) =/»(/»;/«') (3) . 

/«,(p'":p.'4'".ri) = /„C/';p./i".ri) (4) 
b*>Li^*te.. A7;*-^A'...A"43J:o:r.tt*»Ta&D. I®lli<D^M©*fJS'b 



^£ 2002-515150 



'<TCDp'©V1^9fiaiffi-C»0. mp){t Z. (D J: o U-r <Dp' <D t) > 
^±iH^iS:ft©iRfSiJ;>5:'y — XT?*-5, 

SC(5)ic*3it«SjgftBg»;5i&afJciEgliC(i)*iJ;r^(2)*^6. /-^^ >c 

- ^' * EB i; it (closed f orm)T? » % C t *i T ^ V» d t « ?q 5 *^ T a& « . LM 

g((^'':p);M.y, )= ' (/,(;>' )-/(/>)). (6) 
&n*<!:, ^-n^fn WE<(p' :p) :A. T . SC(7 n« <k 5 d 



( 12 ) 

'v;^ S (map)ili » rtO-iffl(D*f^f^«)^*0^«!tSrg/hfkU, 

min I . - _ 



( 13 ) 0 0 2-5 1 5 1 

Eiip' ; p); A.,, ) - EUp' : p); . r,, ) + ' 

E((p' ; p); A, . r,. ) = -j=^ l. ip'ip: .7,, )) - Al .7,. )) (7) 

±a:tc;fettS*a^-««ti:ltp'lc*3t»T-9->:/>»U$nfciiilfei<D;t:«i)©5Si« 

7 — Ak T ffl t» /i: 19 * V «y bf > i/^ (forward mapp i ng) T? & -S . (p',p)0 

5. Ase<}tc. P7i^e,p'®afi:fyv-ytf>i/©i5<.4)-*^n«. m — m\t. Mmv&miz 

^\'^xm-^n^u-z/'tstitL¥mz^^T^<^ti!>i-v^, -rtatj^ . i.(p'(p:A. 
'.r,.)) = i.'(p)-r$>-5, ^ (o - zr tiftm\t . p*iJ;yJ/x^;< — :^A»'.r.k* 

ffl V> X mil! i tC O t» T CDp' *ti-»L. -e-tl^i&MESSce^-fe^K^ecBTOlltl.CDgt. 

/t ^ > - ^ M tc ;te t/s TE ( (p' , p) : A» ' . r ■ k ) = ii • (p)-l' (p) T * ^ . 

-j^Z v/f (p' )|/' V.4.n^ , Vp':|(p; Ma .ri,)) (8) 



( 14 ) 4$S2 002-5151 50 

a<D ^ - ^ i; h (D'Atc. H\tm»<oik. Wt^mcD X :h ^ - r I ^ 

o 

mmmm<D^Si\it , iy.T©J:p»c*-rctjjiT^^:v(A, .t.)p'=va'p,'+v 

(A' . T . ) r (p, ' : r . ). ^3 BH tt a' g| T s fe o t , r , iz mr ^ h <o ii ^ ^ ti 

SC(3)d^e., VA'p'=((l + T . r')I,+2r . [p,'-Pc'][p,'-Pc']')V lA'p.'T* 
0, ClC:TI,(i2X2K»g^TH'Ju/i:7XT$..5. 

S nfc (augment ed)'^ h ;P p. = [p I ] ' S: ffl T , (9) ® » il fiR S tl 
. ZL h U y ;^A' = [A, " A, " A, " ]' T » S . 



.r ,r ./^ 



Mj'^pej Af pa A^J pa 



4* /Mt 



(9) 



^ ^ tC. V r , p' = [p, ' -Pc ' ] T 0 . g' =V Ii (p' ) V A' p' ^ 1 X M-T h U ^ 

r ' =V I. (p' ) V r , p't^ 7. * ^ - <h r ^ o r (p) = [r ' . . . r * . . . r ] 

t*'<TOr'01XM*NVhU>:/^Xi:U. g = Vir'tT^o ■r^<i:> SC(8)OVE 

ZpZ/(£(p';.p) + -£:(p'; p) ^j)^ 
s«pji5igtciitts-€-n-en®^pjc*tL.T. pjcs*^*^tyt-'<T©iif<|.dt, 

diagonally dom i nan I ) S: -S CI i (C J: o T , ^ M H Sc © M 'MlS t" -S $ Sli T 

e 



( 15 ) 002-515150 

^ m \^ T mti m \z ^ ^ (O ^ - ^S' ^ m m iz m Wt ^ 1 7b^ T ^ ^ J: o iz t ^ 
/■^ y :^ — 9 ©»!fiji8(|!(coarse-to-fine tracking) 

as it fl<i « 

■S>^8N+lffl*3j;rXl<l<D^^jl©^>XM*/t^^-^)*3a»e<HC»<fc«)IC. 
*Sai'l4 7&<:;^C^IC(progressive. «jte<JI:i)lti)PT-S^7^;i/0<sgffl*ta* Ll^Cl i: 

(5) »c # A ^ n . 8fe < ^ M M d« S /JWb $ n * . M X 5^ r Ji^ & © iiS » X 7=- 
>;/ »i if ffi <!: b T ffl e. n -5 . 

1. — AJc:?liT. 2;^7cMii. 20®*»/l7;<-^'.a,,', a,,'( 
S:i)(;^T->yr3 0 1), 

%—\z. nm<r>^ >h^<^^-Hm . ASM»cttA:^H«fco^i*:7Elc»5i/3. 

:^ m iz is if i> \d i/ -t }]/ ii . n 615 © ^ 1^ ic tt -5 e i7 -fe t it T « i ^ i* U > X 
S * & S It iii . 

2. T7-C>Slft, 6 7 ^ - ^^a, , ' . a,,', a,.', a,,', a,,', a, 



( 16 ) !|#S2 002-515150 

(xy" y y 3 0 2 ), mm 7^ 7- y y ^ (D V} m w m — f&<o 7 y ^ >^»/-^ 7 

3. ^ (I) a J;: U n) iz ;^ ^ J: z) t£ Si m 8 n y - \z ia ^ . if u - n }V I' > X 
^ ^ ^ ^ — {If { ^ -D (D n "7 :^ — ^ <D K U S m.*"^ I tC^L< tS. ^ tfSitlL^W 









) {7. y y ^ 0 3 ), 












m.^\z. St y ^ ~ 3^ t u > X m 


i: 










ffl S , d 


CD 




y T « , K M M 


x:h m <D ± \z ^ It -o X mmit n 


-& 




CD 




X . X ^ 


y y 3 0 2tt?ii«UTjt«e)6.nsj:t*** 


•5 










«F ffi « /J> it 














> 


/ 7 yy> 


7 


> tf 


^ ^ «y H 0 » ffi U 


;nc t3 -5 S a *i . & 4- ® e -t ;nc 




i^x^&iz 


•D 










y H © ft ift 






JC :fe T ffi $ n 




0 






;i/ CO 




t» e. n s . A s WJc « , 2 0 © «g IS » ^ 


$ 


n * . tf ^ 






0 iS CD u ^ ;i/ 


IC 43 l» T « . A S M IC . ^'A^^y" )V (D ^ 




^ n -5 . 




n ^ 


CO i/- ^ ;i/ e> CO ite 


mufi. e V 5 .y F CO » ^ ;i/ fc *$ T iS 




































^^mmmmiStiz-oi,^x<D:^^m<D!t^mo 




« , rjJ B5 


(D 














t \z 


^ 0 dtl e> CO ;<7 






« il (c iS 






^ 0 c n e> CO * 






T # S ^ 






^ e6 {C . 1/ > X 






<!: a ^ L 






u-^iD^^m^^^Vim<twtm\z^\f^x . 




y V' — A\Z 




T 


, » ^ lUi S ?iJ /t 


y /.-if (onwt tt\zu >x^^'&s^n-r 


* * »C ffl 




s n s 0 



( 17 ) i^«2 002-515150 

^ — ©09Tt±. @4~7 ^-AOg»g©iBt#a?^t 
(room sequence)*^ ffl ^ n ^ . C ® ^ ^ X v SS^<D(hand he 1 d) iS ffi T ;i 

S:^T;fc:«S)Jc. 7nJ;0'b:*:#;^c7C'-Alc^$txTiriS. 06 tt. 71^ — AOtc. 

JiSffi-rjSiti * Sb O- ® » (gooseneck, tt L U 0 # tt A * l# :3) * * J>t 7 Sr 
ffll»T3tac$nfcK^3.;*>h<D30©|fe*t. 08JC«3nTl»-5. HISSES 

g-r«C:i:*<T^«. 09 tt, tH¥^nfcU>XS**3j;ytyp-/N-;H#^/'^7 

±tC||fI$n;tT>»l/zf<J XA«. -^;^SnfciSli*-S2p7 U-AJC*JViX7 U 
-A<D-r'^TS:fB©t--6fcJ{), U-AtCt>«:-3TI^«PIC®fflSn-5 C t 



( 18 ) 002-515150 



ifi if, i, 










mis. 




it «) fc 


. m t 




S — 










jc m 


6> n ^ 


I— 


t 




•5 








© t> <D 


tc 






n 


•5 


3^ t *i & -5 . 








/ - 






• 








n 












*5 <ri T 










• 








^ $ n 










*^ 


















, <t V» ^ <»: T? * . JlcDi&^tt. J!2: 7jc(expanded) 


■tt » o 








> 


-r 








$ nfc 




. i 


ft 














n s 












^ o T 




fc 2 


|5) 









0. 1 1 {c^sn-SAMM^T'n-fexic: 


J: o 


T 








■5 . 




1 1 tc ^ $ ti S J;^ ic. 






/I 




> i U T S « n . (f 'L- 








lb 


IC © tf '5 ffl I'* * t 






# $ 


© 




X X 








> © < o © IS lit *^ » 


* >^ 7 $ 



^ — ic, fi6nfcffliis(0i iics^$nfcaift©i-r))©2o*fctt3:3©7i< 



( 19 ) 4#«2 0 02-515150 

(o m =e ji' ^ m X ± <D m m ^ ^mr ^ "5 \z m 

m^ntc, SC( 5 ) CO g jgft: IC fijffl $ n (plugged)M i^c -5 ^ X Jl/ 

. 01 ij±*fc. ^&+i^>xm^mmn7y^-i'^m\'^x'7-yi^ftm(oi-:> 





i^^fiffi RMSMM 








6 4 > h 


3 27l^'f > h 


3 2n'^'< >h 












1.36 


1.17 


1.56 




0.67 


0.64 


0.72 


7 7 >+LD 


0.60 


0.57 


0.46 




0.26 


0.39 


0.34 



L D > X3i *.) 

, ■!t-rX 3 2 0. 2 4 0C0ia^tCOl.iT1 6 0. 1 2 0T^-5<h^3^SnTl>'5 



{ 20 ) !RfS2 002-515150 

n it #^ 7 U - A (7 k - A 1) i , (te(D2t><7)7U-A, 7 W - A 0 *5 J: 7 
-A2 i:©W©^JC-:>tiTORMSKII«:«ftT-5. S'bSliffFffiK^«(160.1 
20)*3 J; (170, 1 1 0)<D fit IC O T ± U , •1' t O 7 - ^ i CD M :^ S: S » 

-r-5*:«6JC. ^ Jj: f -5 ^iKpoint estimation er rors)-C ;^ < . ft /Jn ^ « 





V b — A 0- 1 RMSI^M 


y u — A 1-2 RMS^^ 


X 


Y 


6 4 7t?'<> V 


6 47^^" > h 


tr ^ -fe ;i/ 








160 


120 


0.261 


0.289 


155 


120 


.0.277 


0.337 


165 


120 


0.277 


0.307 


160 


115 


0.273 


0.297 


160 


125 


0.355 


0.310 


170 


110 


0.292 


0.242 


170 


130 . 


0.387 


0.404 


150 


130 


0.320 


0.358 


150 


110 


0.420 


0.430 


160 


115 


0.273 


0.297 


180 


100 


0.363 


0.320 


190 


90 


0.344 


0.353 


200 


80 


0.395 


0.418 



( 21 ) 002-5151 50 

tt. •?-<D7^/'C-f;^(7. + -v:^-*JJ:tX/ifc«ia«^S)G)jgJtHtt&*«Kt--5!^SiJ/j: 
S(flat bed) 7. ^ -r \t:^ri E ^^-r i> fztb IZ K ^ Z. Hi -V ^ T . mW(OX 

+ -v:*-»±itjR*x=^^-v>u/to. m-Bi^M^mi^tzor^rci()\zm^^i>z.t\fv 

€r . 

5>JBIfe © S ^ (blendlng)IC » l» T i/i -5 . C CD S 5iJ « . 'e-^'fi7f)iiffS.^n^t 
— AtJ^ U- — A'^<D(fraiDe-lo-consecutive-fraine)||5'i*5J:C^ 1 ^^CTCjg-o- 

zfWM^oDj^mtzisi-^Tit. &m a xz/ 2 'AtLx <om-Bt a^isi ^Rumti m ^ 
0i2£#RgT«t, :/Plr;^(i 2 0 0)'^oA:^iai»7i<. ii<o*»(Diiilfeft 



( 22 ) «fS 2002-5151 50 

T # -5 . 

\^ > Xm^Wm/WiEiT.'r 1 2 1 OtsJiZfl 2 2 0) 

GiMmx $> 0 m 1 2 0)7.7- y y I 2 1 O). 'iJSt$nst'©4^iH(7.5^-;/ri 

e 

i: *t T # . 

71^ — Z>;&^&7l^-A^(Dgg^J(7l^ — A-7U-AMS3^J. Xt^^T^I 2 3 0) 

, ctitt, 2 -T) ojiigi: 7 u - A CD « H © * s SB^j- * !B5ij -r s 2 j*:7cMai(fj: e 

#JS«t'*5 J; UCfilf ^rcDtUam 6 0 / 0 2 1 , 9 2 5 # *J J: ^ 0 8 / 3 3 9 , 4 9 
■5.*fflW^^08/328. 7 8 3 ^ fiSYSTEM 

AND METHOD FOR ELECTRONIC IMAGE STABILIZATION) \t . /"^ — 3^ M m (D 0 A 



( 23 ) 2002-5151 50 

W Sr <t 0 ff # JK e. -B- «> (spread oui) l- - ^ )[^ . S t ^ if 2 ffi 

J: IX 1 2 5 0 ) 

nco/i5;< — ^m<nrf - ^ y if f^f-^ i7 o) -n ^xsy (D-x^ ^m^-t 
^ itisb \z m <b n ^ m 1 3 (o y i 3 1 o. d©"? — ^^y^-tF-f^^tt. 
2|E5!w©siJgo?^^tcagoT, XT^-yT'i 2 3 ojc*5v»T»e.nstn¥$nfcs 

^s?j»ift$ns(x5^«y:/i 3 2 o). -e-ciT< OigL:^D-irx*ii6*^. frbvi^ 

4'i:|^i;-y-'fX**L^tfi'-fejHc-:3liT#^/«l2!i;S:«li(status). tts.f^% 

(XT--y:ri 3 3 0). tfi7-fe.;|/?£-^tJll^C07^-A<£MltecD'7 — =*^>i7t+f'f 

. Ld^ L i; 0 -9S:e<) fc « . 7 7 ^ > *3 .J; « :^ /t 7 - ^' ^ ffl T , <S S © 7 U 
— A'N8ij!l$ns(:;<.T-5/:/i 3 4 o). c:©S?ijft$g©fi^«BCoviTtt. *Hm 

Bm60/021. 9 25^:^08/339. 491#:S08/328, 7 



( 24 ) 0 0 2- 5 1 5 1 

8 3 mi JiXfm 0 8/4 9 3. 6 3 2#IC7K$nS. *5£W®IISS©}^fii«c:(i£r) 

':/yi 3 5 0). yP-feX«. ■r'<T®7U-A*ig|e>7i<JiC^*T, s-^u- 
AJwOt>T»Djg$nS(:^x-;/yi 3 6 0). 

2:;^i:7c^tf'f^tc*fUTr'<T©7U-A*S«IMtcSE5>J-r«fcJi)t;i, 
X ^ -tf -r t & ffl Tffi it M JCili 0 il 3 n -5 d t *rT # -5 . ^ti«J);^. CO r 

2;^7t:jB^(:^5=-y7"l 2 6 0) 



7 1^ — 






if 'f CD 


tr i7 -fe 




ft o^ft 






M IC J; < 








set 




Ac ^ ^ /t 


7 pi - 


\z J: X i&m^ ni> y 


* ^ ^ B 




u iis iB e <D m * ^ if 'f 












— A«rt|-^-r«.. 



© 0S ©fg/ft © >S: * . 7^-A*<^M^ 
-A(Dift|fMJ::SJc*^SffllJt©t5#B 
}^fii©S^(blending)ft^l5tt, th fit ^ n 
U — AWhJt^Dv^-SrffifflL. 71^ — A 

je ^-r * . 
■tf <f i7 tf i7 -fe ;u »c o T , tr -fe 

© »i ^ « > A S M tc , Hi 



»©«^^*^cjavifc:^'■fe;^J5tS/^©2*■&'^)t^^feiai^'»l¥im*^^T^»st^»■5te 

<g^(Voronoi I esse 1 1 a t i on)*^ , -tJ- h i: L T =& H # 7 U - A © ^ fife $ 7^: 4^ 'L« 
^ — -> 3 > ;^ 5 7 (mot ion snear) * « H » fil « © ffe © ^^ fl: , JC J: -3 T iffl 



( 25 ) 2002-5151 

m<D\,^ < o*^ji<^s«r§its«^tc«. z. ti ^ (D m fz it ^ <o m s ^ Si i-t tz 
n*^ <a mtE<r>m»tii \z96 K> ^ ti^ . <d & m '& ts m^E (o m m \z >b 

if^^x^r— hvx. ^ly^jntmm^n, ^ (d u ^)v\ziin ^ ^ =y i^r y a ^ 
O y 7 > ^E^^f -i i7 . 'A<Di'^}V^^f$.-r ^tz>i>\,zlsi7L^ni> . c:<z>:/D-tr 

□ -bXCDfifiBUCHbTtt. *@WH^6 0/ 0 2 1. 9 2 5#:^0 8/ 3 3 9 
. 4 9 1^:teJ;r^^0 8/ 3 2 8. TSS^lC^l^nXt^S.^^fcCtlf-fi^cD 

T ^ ife ^ n f# , 

d)yD-t:x*«ki;c^n&cD7'D-lr;^*Slff-r4^e<ojgit:icJ: 

■D X tin ^ . 2^5£e^tt*fc, 7P«>e-7^-<y^i7, cd-rom, 

feints. |2&«S#:t«#$n. n>t!3.-^'tCD — h'$n*JJ:i;c/ 



( 26 ) 2002-5151 50 

fc «i fi)t -r S . 



( 27 ) 



4**2 0 0 2-5 




"T 

16 I 

I 



( 28 ) 



#^2 002-5151 50 




( 29 ) 



( 30 ) 



2002-5151 50 



[@ 4 ] 

FIG. 4 




( 32 ) 



2 0 0 2-5 



[0 6] 

HG.6 




( 33 ) 



#a 2002-515150 




FIG. 10 



[08] 



( 34 ) 



#^2 002-515150 





FIG.8 



( 35 ) 



002-515150 





FIG. 9 



( 36 ) 



i^^2 002-5151 50 



FIG. 11 




( 37 ) 



i^S2 002-515150 



[0121 

m 1 2 



1220 




Sift/ex:!- 



1210 



1230 



1200 



1240 



1260 



1260 



( 38 ) 



2 0 0 2- 5 1 



7 b— -7— 



■p \y—M, • 7 b— i*(HI 



.1310 



,1320 




1360 



iSu:? b— A$ 



.1330 



saa>7 b— A^fiEJU-r-s 



-1340 



aS(D7b— A* 



,1350 



( 39 ) 002-515150 



INTERNATIONAL SEARCH REPORT 


felODstiooal MppUcmhoc No. 




PCr/US97/20762 


A. CLASSIFICATION OF SUBJBCT MATTER 




IPC(6) <30&K 9/32. 9/36 




US CL : 2%2QM, 394 




AcoonSiof to IflUfBAltQttiJ Pdont CUuIlk«xiM (IPC) or lo both oatioasl drufficiioa ud IPC 


a FIELDS fiBARCHED 



MiaioMim doouDOsMMo «oo m b»d <ol«*fiSGStioo fystMB foliavod by cknificAlioii synboli) 
UJ. : 3S3y3ft4.294 



Docsncotetion mschckl otbov tb«i mSnimuM doaUBfiatetioo to Ibe B3Ctoaill»l vacb doevncatl of* inotuisd n tibo fiolds woiebMJ 



Etoolroolo tfiift bMO coftwlietf diitas <he taKiurioul seaicb Owmo of doto fauo «iid. wbcic iiracticobk* •earob temu iMod) 
1EEE» STN» INTERNET 



DOCUMENTS CONSIDERED TO BB RELEVANT 



Caugoty* 


Ciuiioo of docvncoc wi4 iadio«tioa» wbere ftpproprioiB, of (be rt levant pgiMgn 


Relevost to dauai No. 


X 


KUMAR, et al. Representation of Scenes From CoHectkni of 
Images, IEEE, 1995 pages 10-17; see page 14, sections 4.1 and 4.2; 


1.5.6 


Y 


see entire document. 


2-4. 7-13 


Y 


C. F. MCLEAN Image Warping for Calibenuion and Removal of 
Leas Distortion, IEEE, May 1993, pages ITO-ITS, especially page 
171 


13 


Y 


SAWHNEY, et al. Model-Baesd 2D&3D Dominant Motion 
Estimation for Mosaidngand Video Representation, IEEE, aUGUST 
1995, sec entire document, pages 583-590. 


1-13 


Y 


IRANI, et al. Mosaic Based Representations of Video Sequences and 
Their Applications, IEEE, August 199S, see entire document, pages 
605-611. 


1-13 



X| Further doeatn«ou mo in(t»d in (be ooaiiBuatiaB of 8ok C. | | See palm I fajaUy aoacx 



6»mmi m ^t 6ttaing g^armJ mtmU a/ lha art wfcub is am MnMJ«Mi 
U b* of pmni mt m ntovmM* 

nrlier Jocmtnin pabGiM oa or mfw ik* wmw b i ^l fSa^ dM 
4oean«at whid) B«)r ihfiNr 4oukt» od t«iarKy chtii(i) or witkk ii 



A«» «od o«i ia Mettc« wMi tfa* ByHiraiio b«l ciiad lo 

d0cnii«ni af pwtloukr nl«v«nM: tiie cWRMd invwitioa 
caaMdrnd Dovsl or cnaiMbt aiMiidnid la iBvokra ■« 



aocMfltm publahBd prior ID Ika a 



dactiA«ai af nawirtibr ri W » aum : ihft duawd wimhi 
oos9Bi4«-«l •» AbvoIvo n braaotivt lUp urbM tho 

teiaf obviau* I* • pMiaa ifcOkd « ib» w 

d »W Mii voibW of lb* aw* pMBOl 



OaiB of tbc actual complolioa of tb« ialenalioaol aaareh 
10 FEBRUARY 1998 


Dale of aiftiliag of die intematiooai ceaicb report 

/ .11 MAR 1998 


Name and fnailtng addrcis of the ISAAJS 
OoavmsMOAcr arPalenlt nd TrademAriet 

Box per 

Wwhfal^ D.C 20Z3I 
Faaaiietla No. (703) 303-3230 


pAHlELQ, MARIAM 
Tdepbooo No. f703) 305-4010 



Foimi FCT/ISA/210 (second sbeelXittly 1992)* 



( 40 ) 



2002-5151 50 



INTERNATIONAL SEARCH REPORT 



bteti>tioB«t ftpplicalioa No. 
PCTAJS97M762 



C (CostiDumtiini). DOCUMEKTS CONSIDERED TO BE RELEVAHT 



CtCogoiy* 



Ckfttkn oT doeanat. willi mdiuticM, where appcopfUto, of Ibe roleviBt pMMgM 



IUl0v«Dt to eina Ko. 



US 5,566JZ51 A (HANNA et al) IS OCTOBER 1996, see endue 
document 

US 5.187,754 A (CURRIN et al) 16 February 1993. see entire 
document 



1-13 



1-13 



fan PCT/ISAyZiO (conti]ittatio& of second shcelXJuty 1992)* 



( 41 ) 



002-515150 



7D> h'<— >*Oi6S# 

(8l)*tJ£H EP(AT. BE. CH. DE, 

DK, ES. FI, FR. GB, GR. IE. IT. L 
U. MC, NL. PT. SE). BR, CA. CN. J 
P. KR. MX 

atmmm n-fy, ^x-fA^.. 5.y-fe;u 

7;>C'J*^mB -3.- v'-v-v^-iW 
;^h'J-h 57 
?iSCD^»cr)fDT-a&^^|ga (error measure) /O^^/hft 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the appHcant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SffiES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENC£(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem MaUbox. 



